[SPARK-59410][SQL] Derive PartitionPredicate from identity fields of a mixed partitioning - #58702
Open
pan3793 wants to merge 1 commit into
Open
[SPARK-59410][SQL] Derive PartitionPredicate from identity fields of a mixed partitioning#58702pan3793 wants to merge 1 commit into
pan3793 wants to merge 1 commit into
Conversation
…a mixed partitioning PushDownUtils.getPartitionPredicateSchema returned a schema only when every transform in Table.partitioning() is an identity transform, so a table partitioned by e.g. dt (identity) and bucket(16, user_id) never received a PartitionPredicate, and a Catalyst-only filter on dt such as the cast(dt AS DATE) = DATE'...' produced by type coercion could not prune partitions in the static pass, via DPP, or in a metadata-only DELETE. The schema now has one field per transform, in partitioning order. Identity fields carry an attribute a filter can reference; other fields have none but keep their ordinal, so a predicate still binds against the full partition key and the connector contract is unchanged. A filter on the source column of a non-identity transform stays a data filter. A partitioning with no identity transform still yields no schema. The in-memory V2 filter test table now accepts only column-vs-literal predicates and returns anything else, e.g. a predicate over a cast, as a real connector would. Assisted-by: Claude Fable 5.1
Member
Author
|
cc @szehon-ho |
uros-b
reviewed
Sep 11, 2026
uros-b
left a comment
uros-b
left a comment
Member
There was a problem hiding this comment.
Looks clean to me, thank you @pan3793! But yeah let's definitely add @szehon-ho who has more context in PartitionPredicate
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
PushDownUtils.getPartitionPredicateSchemareturned a schema only when every transform inTable.partitioning()is an identity transform. It now accepts any partitioning with at leastone identity transform: identity fields can be referenced by a
PartitionPredicate, otherfields keep their ordinal but are never referenced, so a filter on the source column of a
non-identity transform stays a data filter.
The connector contract is unchanged:
PartitionPredicate.evalstill receives the fullpartition key, and
references()still reports ordinals intoTable.partitioning().This applies to all three users of the schema: the static second pass, runtime filter
pushdown (DPP and scalar subqueries), and the metadata-only DELETE rewrite.
The in-memory V2 filter test table now accepts only predicates of the shape it can evaluate,
a column against a literal, and returns anything else, e.g. a predicate over a cast, as a
real connector would.
Why are the changes needed?
A table partitioned by, for example,
dt STRING(identity) andbucket(16, user_id)receivedno
PartitionPredicateat all. A filter such asdt = DATE'2026-09-01'is analyzed ascast(dt AS DATE) = DATE'2026-09-01'; a connector that does not evaluate casts returns itfrom the first pass (and without ANSI mode it is not translatable at all), so only a
PartitionPredicatecan prune with it. On such a table it could not prune partitions ineither the static or the runtime path, and could not drive a metadata-only DELETE, while the
same filter on an identity-only table can. Mixed partitionings are the common case for
connectors that support partition transforms.
The all-identity check was raised in the SPARK-55596 review (#54459) and kept only because
the partition-key contract was still open then. The contract that shipped (full key,
partitioning order) is what makes lifting the check safe.
Does this PR introduce any user-facing change?
No. A connector that opts into iterative pushdown and has a mixed partitioning now receives
PartitionPredicates over its identity fields; theevalcontract is unchanged.How was this patch tested?
New tests on a partitioning of one identity column plus a bucket transform: a predicate on
the identity field is pushed and prunes in the static pass, via DPP, and in a metadata-only
DELETE, including the
dt STRINGvsDATEliteral case whose type-coercion cast the sourcecannot evaluate; an identity field placed after the bucket transform binds to its own
ordinal; a filter on the bucket source column is not turned into a predicate; a bucket-only
partitioning yields no predicate; the predicate survives Java and Kryo round-trips.
Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Fable 5.1